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DETAILED ACTION 

1 . This is a Non-Final rejection in response to amendments/remarks/RCE filed 5-3-2006. 

2. Claims 1, 4-5, 8-13, 16-17, 20-25, 28-29, 32-36, and 44-52 are pending in the case. 
Claims 2-3, 6-7, 14-15, 18-19, 26-27, 30-31, and 37-43 are canceled. 

3. Effective filing date is 3-15-2000. 

Continued Examination Under 37 CFR LI 14 

4. A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on 05-03-2006 has been entered. 

Claim Objection 

5. Claim 44 (Currently Amended) of the amendment filed 05-03-2006 was not properly 
marking to show changes (i.e., for deletion of five or fewer consecutive characters, double 
bracket [[ ]] may be used. Any future correspondence must contain the proper marking to show 
changes. (MPEP 37 CFR 1.121) 
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Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 1, 4-5, 8-11, 13, 16-17, 20-23, 25, 28-29, 32-35, and 44-52 are rejected under 
35 U.S.C. 103(a) as being unpatentable over Schena et al. (hereinafter Schena), US 
6,448,979 Bl filed 1/25/1999, in view of Robinson et al. "The Origami Project: Paper 
Interfaces to the World-Wide Web", 

[http://www.cLcam.ac.uk/Research/Origami/Origamil997f/index.html], submitted to 
Webnet 97 in November 1997 (hereinafter Robinson). 

Regarding independent claims 1, 13, and 25, Schena teaches the multimedia 
annotation representing at least one of an audio sound and a video clip, wherein the at least 
one of the audio sound and video clip to be extracted from the multimedia annotation, and 
the at least one extracted audio sound and video clip can be played via a multimedia player, 
and the multimedia annotation represented by a first bar code encoding at least one of an 
audio sound and a video clip, wherein the at least one of the audio sound and video clip 
Specifically, Schena discloses the multimedia annotation such that the multimedia annotation can 
be extracted and decoded subsequently from the first multimedia document, wherein the scanner 
detects and reads data, such as machine-readable codes containing link information 
corresponding to provider information from the printed medium. The link information 
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corresponding to the provider information may include, a universal resource locator (URL), such 
as customer premises equipment (CPE) for displaying the multimedia sequence information. The 
multimedia sequence information may be advertising or transaction information and may contain 
one or more of textual, audio, or video information, and the customer premises equipment 
("CPE"), serving as the receiver, plays multimedia sequence (Schena col.l, line 64 through col. 
2. line 40). In addition, Schena discloses the scanner is capable of reading data such as non- 
coded data and machine-readable code 10 from the printed medium 50. The machine-readable 
code 10 (URL) may be a barcode, an enhanced barcode, a new enhanced code, or any type of 
code, including dynamic codes and high-density barcodes (Schena col. 3, lines 55-65). 

The Examiner equates the claimed the multimedia annotation representing at least 
one of an audio sound and a video clip as equivalent to the link information may contain one or 
more of textual, audio, or video information as taught by Schena, and the claimed a multimedia 
as equivalent to the plays multimedia sequence as taught by Schena. Also, the Examiner equates 
the claimed bar code encoding at least one of an audio sound and a video clip as equivalent to 
the link information from barcode, code 10 (URL) may contain one or more of textual, audio, or 
video information as taught by Schena, and because the Applicant invention specification 
discloses "The annotation may be in different forms, such as, for example, a bar code containing 
an audio message or a URL indicating a link to a video clip (Applicants invention 
specification page 7 lines 15-18). 

In addition, Schena does not explicitly teach, but Robinson teaches creating a 
multimedia annotation for a paper document, and creating a first multimedia document by 
combining the paper document and the multimedia annotation, wherein the first 
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multimedia document is generated as a part of reproducing the paper document via a 
document reproduction system, and wherein the first multimedia document, which when 
scanned by a process, the process causes the printed multimedia annotation to be decoded. 

Specifically, Robinson discloses the origami project, wherein the printed documents are 
annotated with marks in their corners to facilitate recognition and location on the desk top, and 
are also have a unique identifier printed in an OCR fount (see Robinson page 4 section Printing), 
these steps involve recognizing that a page printed by the system has appeared on the desk, 
determining its position, reading its unique identifier and locating any interactors. A 
transformation is then set up between the page representation stored in the registry and physical 
co-ordinates on the desktop. The printed document thus becomes part of the projected window 
system, wherein any active links are highlighted by projecting a red background over them. For a 
document originating on the Web, these correspond to links in the original HTML (Robinson 
page 4 section DigitalDesk, also see fig. 2 a-c, as example of the process of annotating a printed 
media). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Schena to include a method for creating a multimedia annotation for a paper 
document, and creating a first multimedia document by combining the paper document and the 
multimedia annotation, wherein the first multimedia document is generated as a part of 
reproducing the paper document via a document reproduction system, and wherein the first 
multimedia document, which when scanned by a process, the process causes the printed 
multimedia annotation to be decoded as taught by Robinson. One of ordinary skill in the art 
would have been motivated to modify this combination, because Schena and Robinson are from 
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the same field of endeavor of providing multimedia annotation for a paper document and for the 
advantage of bridging the gap between the virtual multimedia-based of the Internet word and the 
physical world of printed media, wherein the annotate media may contain one or more of textual, 
audio, or video information, and the customer premises equipment ("CPE"), serving as the 
receiver, plays multimedia sequence (Schena col.l, lines 5-15, col. 1 line 64 through col. 2. line 
40). 
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(c) Animated on the DigitalDesk. 
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Regarding claims 4, 16, and 28, Schena teaches wherein a location indicator 
associated with the multimedia annotation is placed on the first multimedia document, 
wherein the location indicator indicates where the multimedia annotation can be retrieved 
and played. Specifically, Schena discloses the link information corresponding to the provider 
information may include, a universal resource locator (URL), such as customer premises 
equipment (CPE) for displaying the multimedia sequence information. The multimedia sequence 
information may be advertising or transaction information and may contain one or more of 
textual, audio, or video information, and the customer premises equipment ("CPE"), serving as 
the receiver, plays multimedia sequence (Schena col.l, line 64 through col. 2. line 40). 

Regarding claims 5, 17, and 29, Schena teaches wherein the location indicator 
comprises a first Uniform Resource Locator (URL), and a second bar code, wherein the 
first URL is indicated in plain text, and wherein the second bar code represents the first 
URL in an encrypted form. Specifically, Schena discloses the link information corresponding 
to the provider information may include, a universal resource locator (URL) (Schena col.l, line 
64 through col. 2. line 40). In addition, the links can be encoded according to provider, for 
example UPC or ISBN numbers and any code may serve as the alphanumeric sequence (Schena 
col. 4, lines 45-55). 

The Examiner equates the claimed bar code as equivalent to the UPC or ISBN numbers 
and any code as taught by Schena, and because the Applicant invention specification discloses 
"The annotation may be in different forms, such as, for example, a bar code containing an audio 
message or a URL indicating a link to a video clip. " (Applicants invention specification page 7 
lines 15-18). 
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Regarding claims 8, 20, and 32, the rejection of claim 1 is fully incorporated. In 
addition, Schena does not explicitly teach, but Robinson teaches generating an image of the 
paper document, the image of the paper document being unconsciously captured via the 
document reproduction system during the reproduction of the paper document without 
user intervention. Specifically, Robinson discloses the animated paper involves scanning image 
of the page from top to bottom, left to right and emitting text or images as appropriate. One of 
these is for the HTML of the page itself (Robinson page 4 the Export section). 

The Examiner equates the claimed unconsciously and without user intervention as 
equivalent to the animated paper involves scanning image of the page itself as taught by 
Robinson. 

Also, Robison teaches storing the image of the paper document and the multimedia 
annotation in a storage, wherein the second multimedia document is an electronic 
document associated with the first multimedia document, which is a physical document 

Specifically, Robinson discloses a Registry, which maintains the association between electronic 
documents and their printed variants. It stores the image of each active document and the code of 
any interactions required for the document, together with cross references between these and 
further indexes to identify them (Robison page 3 the Registry section, also see Fig. 1). 
It would have been obvious to one of ordinary skill in the art at the time of the invention to 
modify Schena to include a method for generating an image of the paper document, the image of 
the paper document being unconsciously captured via the document reproduction system during 
the reproduction of the paper document without user intervention, and storing the image of the 
paper document and the multimedia annotation in a storage, wherein the second multimedia 



Application/Control Number: 09/526,03 1 Page 1 1 

Art Unit: 2176 

document is an electronic document associated with the first multimedia document, which is a 
physical document as taught by Robinson. One of ordinary skill in the art would have been 
motivated to modify this combination, because Schena and Robinson are from the same field of 
endeavor of providing multimedia annotation for a paper document and for the advantage of 
bridging the gap between the virtual multimedia-based of the Internet word and the physical 
world of printed media (Schena col.l, lines 5-15, col. 1 line 64 through col. 2. line 40), and 
allowing the database to be built and edited, imported and exported to other forms of hypertext, 
and for documents to be printed and animated on a DigitalDesk (Robison page 32 the Registry 
section). 




Desk surface . 



Figure 1: Animated paper document framework. 
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Regarding claims 9, 21, and 33, Schena teaches wherein a multimedia document is 
represented as a second Uniform Resource Locator (URL) printed on the first multimedia 
document, and wherein the image of the paper document and the multimedia annotation is 
accessed with the second URL (Schena col. 1 line 61 - col. 2 line 25 and col. 3 line 64 - col. 4 
line 6). 

Regarding claims 10, 22, and 34, Schena teaches wherein a third bar code is used to 
represent a second URL (Schena in col. 3 lines 58-63). 

Regarding claims 11, 23, and 35, Schena teaches automatically sending a second 
multimedia document to a recipient by electronic mail as a part of reproducing the paper 
document via the document reproduction system, wherein the recipient is specified by a 
user via an interface of the document reproduction system when the user reproduces the 
paper document using the document reproduction system. Specifically, Schena discloses the 
link information was published or located, along with message-specific information (Schena col. 
4 line 35). 

The examiner equates the claimed automatically sending a second multimedia 
document to a recipient by electronic mail as equivalent to link information was published or 
located, along with a message as taught by Schena. 

Regarding claim 44, Schena teaches wherein the first multimedia document is a 
physical document having the first bar code printed thereon, which when scanned by a 
scanning device, causes the first bar code to be decoded and the audio sound to be 
extracted from the first bar code, and wherein the extracted audio sound is capable of 
being played via an audio player. Specifically, Schena discloses the multimedia annotation 
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such that the multimedia annotation can be extracted and decoded subsequently from the first 
multimedia document, wherein the scanner detects and reads data, such as machine-readable 
codes containing link information corresponding to provider information from the printed 
medium. The link information corresponding to the provider information may include, a 
universal resource locator (URL), such as customer premises equipment (CPE) for displaying the 
multimedia sequence information. The multimedia sequence information may be advertising or 
transaction information and may contain one or more of textual, audio, or video information, and 
the customer premises equipment ("CPE"), serving as the receiver, plays multimedia sequence 
(Schena coll, line 64 through col. 2. line 40). In addition, Schena discloses the scanner is 
capable of reading data such as non-coded data and machine-readable code 10 from the printed 
medium 50. The machine-readable code 10 (URL) may be a barcode, an enhanced barcode, a 
new enhanced code, or any type of code, including dynamic codes and high-density barcodes 
(Schena col. 3, lines 55-65). 

The Examiner equates the claimed bar code as equivalent to the link information from 
barcode, code 10 (URL) may contain one or more of textual, audio, or video information as 
taught by Schena, and because the Applicant invention specification discloses "The annotation 
may be in different forms, such as, for example, a bar code containing an audio message or a 
URL indicating a link to a video clip. " (Applicants invention specification page 7 lines 15-18). 

Regarding claim 45, Schena teaches capturing an audio sound of the multimedia 
annotation from a user using a microphone of the input device to annotate the paper 
document to create a multimedia paper document. Specifically, Schena discloses an 
enhanced electronic device, digital appliances, and microphone (Schena col. 3, lines 40-65). 
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Also, Schena discloses the user interface obtains user input information. The user interface may 
be, for example, a voice-activated system, a keypad, or a keyboard (Schena col. 2, lines 25-45). 

Regarding claim 46, Schena teaches wherein the microphone is automatically 
activated when the user selects a reproduction function of the document reproduction 
system to reproduce the paper document. Specifically, Schena discloses an enhanced 
electronic device, digital appliances, and microphone (Schena col. 3, lines 40-65). Also, Schena 
discloses the user interface obtains user input information. The user interface may be, for 
example, a voice-activated system, a keypad, or a keyboard (Schena col. 2, lines 25-45). 

The examiner equate the microphone is automatically activated as equivalent to a 
voice-activated system as taught by Schena. 

Regarding dependent claim 47, Schena teaches capturing a video clip of the 
multimedia annotation from a user using a video camera of the input device to annotate the 
paper document to create a multimedia paper document. Specifically, Schena discloses an 
enhanced electronic device, digital appliances, and microphone (Schena col. 3, lines 40-65). 
Also, Schena discloses the user interface obtains user input information. The user interface may 
be, for example, a voice-activated system, a keypad, or a keyboard (Schena col. 2, lines 25-45). 

The examiner reads a video camera of the input device as equivalent to digital 
appliances, and microphone as a user interface for obtaining user input information as taught by 
Schena. 

Regarding dependent claim 48, Schena teaches wherein the video camera is 
automatically activated when the user selects a reproduction function of the document 
reproduction system to reproduce the paper document. Specifically, Schena discloses an 
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enhanced electronic device, digital appliances, and microphone (Schena col. 3, lines 40-65). 
Also, Schena discloses the user interface obtains user input information. The user interface may 
be, for example, a voice-activated system, a keypad, or a keyboard (Schena col. 2, lines 25-45). 

The examiner equates the video camera is automatically activated as equivalent to 
digital appliances, and microphone as a user interface for obtaining user input information as 
taught by Schena. 

Regarding dependent claim 49, Schena does not teach in response to a request to 
retrieve a second multimedia document, performing a content-based search for the 
requested multimedia document within the storage based on the content of the multimedia 
annotation associated with the requested multimedia document. Robinson does teach 
parsing and indexing the multimedia documents contained in the registry for retrieval in fig. 1 
and (the "Registry" section in pages 2-3). It would have been obvious to one of ordinary skill in 
the art at the time the invention was made to have combined Robinson into Schena to have 
created the claimed invention. It would have been obvious and desirable to have used the 
registry index of Robinson to have searched and located the appropriate multimedia document to 
retrieve. 

Regarding dependent claims 50-52, Schena does not teach wherein the content-based 
search is performed by OCR, audio speech recognition, or video face recognition 
techniques on the multimedia annotations of the multimedia documents being searched. 

Robinson does teach parsing and indexing the contents of the multimedia documents contained 
in the registry using OCR, audio speech recognition, or video face recognition techniques (in fig. 
1 and the "Registry" section in pages 2-3). It would have been obvious to one of ordinary skill in 
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the art at the time the invention was made to have combined Robinson into Schena to have 
created the claimed invention. It would have been obvious and desirable to have used the 
registry index of Robinson to have searched and located the appropriate multimedia document to 
retrieve. 

8. Claims, 12, 24, and 36 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Schena et al. (hereinafter Schena), US 6,448,979 Bl filed 1/25/1999, in view of 
Robinson et al. "The Origami Project: Paper Interfaces to the World-Wide Web", 
[http://www»cl.cam.ac.uk/Research/Origami/Origamil997f/index.html], submitted to 
Webnet 97 in November 1997 (hereinafter Robinson), further in view of Halliday et al., US 
5,880,740 filed 7/12/1996 (hereinafter Halliday), 

Regarding claims 12, 24, and 36, Schena, and Robinson do not teach, but Halliday 
teaches wherein the recipient receives the image of the paper document and the multimedia 
annotation in the form of Multi-purpose Internet Mail Extension (MIME). Halliday does 
teach sending an image of a document in the form of Multi-purpose Internet Mail 
Extension (MIME). Specifically, Halliday discloses MIME (Halliday col. 8, lines 5-28). 

It would have been obvious and desirable to have used the Multi-purpose Internet Mail 
Extension (MIME) teaching of Halliday to have implemented the automatic sending of the 
electronic multimedia document of Schena in view of Robinson so that the recipient would have 
used any common email client to have received the electronic multimedia document sent from 
the user. 
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Response to Arguments 
9. Applicant's arguments filed 05-03-2006 with RCE have been fully considered but they 
are not persuasive. 

Applicant argues that the combination of Schena, Robinson, and Halliday fail to teach 
combining a paper document with a multimedia annotation (see Remarks page 14- mid). The 
Examiner disagrees. Specifically, Schena discusses in the abstract that the machine-readable 
code on the paper is used to communicate corresponding multimedia information when the 
machine-readable code is read by a scanner. The URL encoded in the machine-readable code 
points to a multimedia file. Thus, the machine-readable code on the printed medium annotates 
the printed medium with the referenced multimedia file, thus combining the printed medium with 
virtual multimedia. In combining the multimedia presentation with the printed medium, Schena 
has created a multimedia document of the claimed invention. Schena discusses in col. 2 lines 6-8 
that the multimedia annotation may contain one or more of textual, audio, or video information. 
Furthermore, Robinson discloses the origami project, wherein the printed documents are 
annotated with marks in their corners to facilitate recognition and location on the desk top, and 
are also have a unique identifier printed in an OCR fount (see Robinson page 4 section Printing), 
these steps involve recognizing that a page printed by the system has appeared on the desk, 
determining its position, reading its unique identifier and locating any interactors. A 
transformation is then set up between the page representation stored in the registry and physical 
co-ordinates on the desktop. The printed document thus becomes part of the projected window 
system, wherein any active links are highlighted by projecting a red background over them. For a 
document originating on the Web, these correspond to links in the original HTML (Robinson 



Application/Control Number: 09/526,031 
Art Unit: 2176 



Page 18 



page 4 section DigitalDesk, also see fig. 2 a-c, as example of the process of annotating a printed 
media). 



NebuayB: Unlvereily uf Cambridge Qjniputtr Lab Hume Page 



HE 



File Edit Vfew Go Bookmarks Options Directory Window 



Help 



ft 

Mnmo 
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Prior 





Location: jiLLp : //www . ul. ucjil. ul . uk/ 





University of Cambridge Computer Lab 



This is the home page at the University of Cambridge Computer Laboratory . Our official World Wide Web 
server is www cl. :am. ac. uk, 

We are the Computer Science Teaching and Research department. Ydu need to look elsewhere fcr information 
ab out the uniyersi-y as a whole, or at out the University Computing Service. 



Local information 

• An overview of the Computer Laboratory 

• List oimernben of the Department, alsa available for TABLE capable browsers 

• Allknawn information server; in Cambridge 

• Cambridge Universirv Information System 

• All np.ws groups available in tVp. local news server 

• Computer Lab newsgroups : 

o ucam.cl. announce, 
o ucaraclrnisc 
u ucam.cl.students 

• Anonymous FTP area for cl. cam. acuk (also available via WWW ) 

• Maps of ths Computer Lab and How to set here 

• Fjid sameone using their Active Badge. There's a Graphical Overview of the Active Badge system. 
(Some of this information is only available to local machines - sorry) 

• Library information 

• The status of the Troian 3,0 om Coffee Machine 

• Usage statistics for this server, updated daihr. 

• Information anc recruitment details for mernb ers of the Ccmputer Laboratory's industrial 
'Supporters -Club' 

• Job v£cancies in the Computer La: oratory 

• Announcements of Computer Laboratory Seminars 



For users in cl . cam .ac .uk only: 

• Local system information page 

• How to export personal ties the WWW 



I 



(a) Original Web page 
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□□□□RbCnnhDDDDDHdnBPH 
University of Cambridge Computer Lab Home Page 
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University of Cambridge Computer Lab 

This ie the home page at the University of Cambridge Computer Laboratory .Our official World Wi 
deWcb server is www.cl.cann.ac.ulr . 

We ate the Computet Science Teaching ahd Research department. You heed to loot elsewhere for i n 
form ation about the university as a whole , or about the University Computi hg Service . 

Local information 



An overview of the Computet Laboratory 

List of members of the Departmeht , also available for TABLE capable browsers 
All Irnown information servers in Cambridge 
Cambridge University Information System 
A LI newsgroups available in the local news server 
Computer L-ob newsgroups: 
. ucam.cl .announce 

ucam.cl .misc 

ucam.cl .students 

Anonymous FTP area for cl.cam.ac.ulr (also available via WWW ) 
Maps of the Computet Lab ahd How to get here 

Fihd someone using their Active Badge . There's a Graphical Overview of the Active Badge e 
ystem. (Same of This &jJbrjncrTion is only available Jo local machmes - sorry) 
Li brary information 

The status of the Troj ah Room Coffee Machi he 
Usage statistics for this server, updated daily. 

Ihfotmatioh and recruitmeht details for members oft he Computer Laboratory's industrial * 
Supporters- Club* o 

Job vacancies i n the Computer Laboratory 
Announcements of Computer Laboratory Seminars 

For users in cLcam.ac.uk 



only: 



Local system information page 

How to export personal files to the WWW 



Documentation 



(b) Printed version. 
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(c) Animated on the DigitalDesk. 
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Furthermore, the Applicant argues that the combination of Schena, Robinson, and 
Halliday fail to teach at least one of an audio clip and a video clip via a bar code that encodes 
the audio and/or video clip (see Remarks page 14 second half and pages 15-16). The Examiner 
disagrees. Specifically, Schena discusses in the abstract that the machine-readable code on the 
paper is used to communicate corresponding multimedia information when the machine-readable 
code is read by a scanner. The URL encoded in the machine-readable code points to a 
multimedia file. Thus, the machine-readable code on the printed medium annotates the printed 
medium with the referenced multimedia file, thus combining the printed medium with virtual 
multimedia. In combining the multimedia presentation with the printed medium, Schena has 
created a multimedia document of the claimed invention. Schena discusses in col. 2 lines 6-8 
that the multimedia annotation may contain one or more of textual, audio, or video information. 

For at least all the above evidence, therefore the Examiner respectfully maintains the 
rejection of claims 1, 4-5, 8-13, 16-17, 20-25, 28-29, 32-36, and 44-52, and should be sustained 
at this time. 

Conclusion 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Quoc A. Tran whose telephone number is 571-272-8664. The 
examiner can normally be reached on 9AM - 5PM EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Herndon R. Heather can be reached on 571-272-4136. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system. Status information for published applications may be 
obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Quoc A. Tran 

Jan. 18, 2007 GAU2176 



Heather R. Herndon 
Supervisory Patent Examiner 
Technology Center 2100 



